…L-20200515-2bug-fix

Feature/update to gfdl 20200515 2bug fix

+(*)Make unit_scale_type arguments non-optional

The original update to allow cf_compilance in diag_manager had the wrong order in the `if` block.

This update reversed the `if` block to allow the namelist do_cf_compliance to work as expected.

Fixes mom-ocean#29

* present_vhbt_or_set_bt_cont: merge couple of loops

* meridional_flux_thickness: cpu optimize a bit

* meridional_flux_adjust: back to jki

* set_merid_BT_cont: pull out meridional_flux_adjst

* set_meridional_BT_cont: optimize cpu

* rm remaining merid/zonal_flux_layer

* zonal_flux_layere: improve naming a bit

* zonal_flux_layere_OBC: improve naming a bit

* improve flux_elem line wraps

* optimize data transfers a bit

* pass elem not arr

* add some missing documentation

* fix trailing spaces and missing var docs

* add last param docs

* meridional_flux_adjust: fix fpe err

* fix another fpe

* cleanup args for new helper subroutines

* clean up enter/exit data

* fix passing h twice

* meridional_mass_flux: do concurrent

* zonal_flux_adjust: use a few 3d tmp arrays to mirror meridional_flux_adjust

* move target update out of continuity

* initialize pbv%por_face_area[U/V] on GPU

* cleanup some transfers from continuity_PPM

* clean up a few minor things

* zonal/meridional_flux_adjust: use scalar u/v_new

* declare vp,up,h_tmp on gpu

* remove h update

* continuity_PPM: minimize mapping stmts

* zonal_flux_adjust: minimize mapping stmts

* set_zonal_bt_cont: minimize mapping stmts

* merional_flux_adjust: minimize mapping stmts

* set_merid_bt_cont: minimize mapping stmts

* zonal/meridional_flux_adjust: tmp vars duhdu/dvhdv 3d -> scalar

* separate alloc of private variables for gcc

* u/vh_aux: 3d->2d

* target teams loop recognizable by amdflang

* Continuity CS outside of init

This moves the Continuity CS to the dycore init function.  For some
reason, this avoids an answer change with CPU.  (Possibly because alloc
inside of a function doesn't quite match the CS outside of it?)

A few minor data transfers are also added to fix up differences in the
chksum log output.

* zonal_mass_flux: isolate zonal_flux_layer

* zonal_mass_flux: seperate local_specified_BC block loop

* zonal_mass_flux: add j dim to tmp vars

* zonal_mass_flux: seperate visc_rem_max init loops

* zonal_mass_flux: seperate du_min/max_CFL init loops

* zonal_mass_flux: separate duhdu/uh_tot_0 init loop

* zonal_mass_flux: separate du_min/max_CFL aggress_adjust update case (untested)

* zonal_mass_flux: separate du_min/max_CFL non-aggress_adjust update case

* zonal_mass_flux: separate du_min/max_CFL non-use_visc_rem update case (untested)

* zonal_mass_flux: separate du_min/max_CFL 0-clamp loop

* zonal_mass_flux: separate do_I local_specified_BC init loop (untested)

* copy zonal_flux_adjust that accepts 2d args

* zonal_mass_flux move j loop into zonal_flux_adjust copy

* zonal_flux_adjust_fused: use 2d internal arr

* zonal_flux_adjusted_fused: separate all loops

* zonal_mass_flux: separate u/du_cor update (untested)

* zonal_mass_flux: replicate former present(uhbt) control flow

* copy set_zonal_BT_cont with 2d args

* zonal_mass_flux: move j-loop into set_zonal_BT_cont_fused

* set_zonal_BT_cont_fused: use zonal_flux_adjust_fused

* set_zonal_BT_cont_fused: separate init loop

* set_zonal_BT_cont_fused: separate duR/L init loop

* set_zonal_BT_cont_fused: separate u_0/L/R init loop

* set_zonal_BT_cont_fused: use zonal_flux_layer_fused

* set_zonal_BT_cont_fused: separate last 2 loops

* copy merid_flux_layer that accepts entire arrs

* merid_flux_layer_fused: separate loops

* merididional_mass_flux: separate local_specified_BC loop (untested)

* merididional_mass_flux: separate tmp variable init loops

* merididional_mass_flux: separate dv_min/max_CFL calc loops

* merididional_mass_flux: separate dv_min/max_CFL 0 clamp loop

* merididional_mass_flux: separate simple_OBC_pt init loop (untested)

* copy meridional_flux_adjust that accepts entire arrs

* meridional_mass_flux: separate meridional_flux_adjust_fused

* meridional_mass_flux: move j-loop into meridional_flux_adjust_fused

* meridional_flux_adjust_fused: separate vh_aux,dvhdv init loops

* meridional_flux_adjust_fused: 1d->2d tmp arrs

* meridional_flux_adjust_fused: separate all arrs

* meridional_mass_flux: separate (d)v_cor assignment loops

* copy set_merid_BT_cont that accepts entire arrs

* meridional_mass_flux: move j-loop into set_merid_BT_cont_fused

* set_merid_BT_cont_fused: use meridional_flux_adjust_fused

* set_merid_BT_cont_fused: separate tmp var init

* set_merid_BT_cont_fused: separate short circuit loop

* set_merid_BT_cont_fused: rm redundant k loop in short circuit

* set_merid_BT_cont_fused: separate dvL/R init loop

* set_merid_BT_cont_fused: make remaining tmp arrs 3d

* set_merid_BT_cont_fused: use merid_flux_layer_fused

* set_merid_BT_cont_fused: separate last loop

* meridional_mass_flux: separate any_simple_OBC loop

* zonal_edge_thickness: move k loop->PPM_reconstruction_x

* meridional_edge_thickness: move k loop->PPM_reconstruction_y

* PPM_reconstruction_x/y: move k loop->PPM_limit_pos

* PPM_reconstruction_x/y: move k loop->PPM_limit_cw84 (untested)

* zonal_BT_mass_flux: separate all loops (untested)

* meridional_BT_mass_flux: separate all loops (untested)

* set_zonal_BT_cont_fused: clean up var defs

* use SZJB_(G) in do_I dclrns in merid* subroutines

* set_merid_BT_cont_fused: clean up var defs

* set_merid/zonal_BT_cont_fused -> set_merid/zonal_BT_cont

* zonal_flux_adjust_fused: clean up var defs

* zonal_flux_adjust_fused -> zonal_flux_adjust

* zonal_flux_layer_fused -> zonal_flux_layer

* meridional_flux_adjust_fused: clean up var defs

* meridional_flux_adjust_fused -> meridional_flux_adjust

* merid_flux_layer_fused: clean up var defs

* merid_flux_layer_fused -> merid_flux_layer

* fix call merid/zonal_flux_layer line conts

* zonal_mass_flux: use visc_rem_u_tmp

* zonal_mass_flux: separate du_min/max_CFL non-use_visc_rem update case (untested)

* copy set_zonal_BT_cont with 2d args

* meridional_mass_flux: use visc_rem_v_tmp

* copy meridional_flux_adjust that accepts entire arrs

* copy set_merid_BT_cont that accepts entire arrs

* set_merid_BT_cont_fused: rm redundant k loop in short circuit

* set_merid_BT_cont_fused: rm redundant k loop in short circuit

* zonal_flux_adjust_fused -> zonal_flux_adjust

* meridional_flux_adjust_fused -> meridional_flux_adjust

* zonal_flux_layer: move GV in var dclrn

* zonal/meridional_mass_flux: remove old visc_rem vars

* remove redundant kloop

* remove problematic omp directives

* port PPM_limit_pos/CW84, PPM_reconstruction_x, zonal_edge_thickness

* PPM_reconstruction_x: add enter/exit data stmts

* continuity_zonal_convergence: save loop range for porting convenience

* port continuity_zonal_convergence

* zonal_mass_flux: array init -> loop init

* zonal_mass_flux: port init loops

* port zonal_flux_layer

* zonal_flux_layer: add enter/exit data stmts

* port zonal_flux_adjust

* port set_zonal_bt_cont

* zonal_flux_adjust: add more arrs in enter/exit data

* set_zonal_BT_cont: add enter/exit data stmts

* port zonal_flux_thickness loops

* zonal_flux_thickness add enter/exit data stmts

* zonal_mass_flux prepare obc for porting

* zonal_mass_flux: merge any_somple_OBC loops

* zonal_mass_flux port main loops

* zonal_mass_flux port OBC loops

* zonal_mass_flux: add enter/exit data stmts

* continuity_ppm: add initial enter/exit data stmts

* port PPM_reconstruction_y, meridional_edge_thickness

* port continuity_merdional_convergence

* ppm_reconstruction_y: add enter/exit data stmts

* port merid_flux_layer

* meridional_flux_adjust: port loops

* meridional_flux_adjust: add enter/exit data stmts

* meridional_mass_flux: port core loops

* meridional_mass_flux: port OBC loops (untested)

* meridional_flux_thickness: port core loops

* meridional_flux_thickness: attempt port OBC loops (untested)

* meridional_flux_thickness: add enter/exit data stmts

* port set_merid_BT_cont loops

* set_merid_bt_cont: add enter/exit data stmts

* meridional_mass_flux: add enter/exit data stmts

* zonal/meridional_mass_flux: add missing vars in enter/exit stmts

* continuity_PPM: complete enter/exit data stmts

* *_edge_thickness: do concurrent

* zonal_mass_flux: do concurrent-ify

* set_zonal_BT_cont: do concurrent-ify

* zonal_flux_adjust: do concurrent-ify

* zonal_flux_layer: do concurrent

* zonal_flux_thickness: do concurrent

* continuity_zonal_convergence: do concurrent

* meridional_mass_flux: do concurrent

* set_merid_bt_cont: do concurrent

* meridional_flux_adjust: do concurrent

* merid_flux_layer: do concurrent

* meridional_flux_thickness: do concurrent

* continuity_merdional_convergence: do concurrent

* formatting

* continuity_PPM: update LB

* meridional/zonal_flux_adjust: separate ij-reduction

* zonal_flux_adjust: a couple jki loops

* set_zonal_bt_cont: some jki loops

* zonal_mass_flux: some jki loops

* optimise loops by using scalar zonal_flux_layer

* zonal_flux_adjust: duhdu -> scalar

* elemental zonal_flux_layere and separated OBC

* zonal_flux_adjust: back to jki

* zonal_flux_thickness: improve cpu perf a bit

* zonal_flux_adjust: add some comments and guard early exit

* zonal_flux_adjust: use omp target loop for private arrs

* set_zonal_BT_cont: use omp target loop for private arrs

* set_zonal_BT_cont: do conc jki loop

* zonal_mass_flux: rm useless do_i init

* zonal_flux_layere: precalc g_dy_Cu*por_face_areaU

* zonal_flux_layere: precalc dh

* zonal_flux_thickness: precalc dh

* zonal_flux_adjust: make uhbt optional

* zonal_flux_thickness: assign to outputs directly

* mv zonal_flux_adjust from set_zonal_bt_cont->zonal_mass_flux

* zonal_mass_flux: reuse visc_rem_u_tmp more

* zonal_mass_flux: remove redundant if stmt

* zonal_mass_flux: force inline zonal_flux_layere

* zonal_flux_layere: inlinable gfortran -O3 but slower for ifort

* zonal_flux_layere: make a bit "smaller"

* zonal_mass_flux: move present(uhbt) or set_BT_cont to new subroutine

* Revert "zonal_flux_layere: make a bit "smaller""

This reverts commit 316152e.

* Revert "zonal_flux_layere: inlinable gfortran -O3 but slower for ifort"

This reverts commit 51ee896.

* fix gcc line truncation

* pass doxygen tests

* remove forceinline dirs

* attempt document vars

* a couple long lines

* last trailing space

* meridional_mass_flux: use zonal_flux_layere

* zonal_flux_layere_OBC: make elemental

* meridional_mass_flux: move dv_min/max_CFL calc into j-loop

* meridional_mass_flux: move big chunk into separate subroutine

* present_vhbt_or_set_bt_cont: merge couple of loops

* meridional_flux_thickness: cpu optimize a bit

* meridional_flux_adjust: back to jki

* set_merid_BT_cont: pull out meridional_flux_adjst

* set_meridional_BT_cont: optimize cpu

* rm remaining merid/zonal_flux_layer

* zonal_flux_layere: improve naming a bit

* zonal_flux_layere_OBC: improve naming a bit

* improve flux_elem line wraps

* optimize data transfers a bit

* pass elem not arr

* add some missing documentation

* fix trailing spaces and missing var docs

* add last param docs

* meridional_flux_adjust: fix fpe err

* fix another fpe

* cleanup args for new helper subroutines

* clean up enter/exit data

* fix passing h twice

* meridional_mass_flux: do concurrent

* zonal_flux_adjust: use a few 3d tmp arrays to mirror meridional_flux_adjust

* move target update out of continuity

* initialize pbv%por_face_area[U/V] on GPU

* cleanup some transfers from continuity_PPM

* clean up a few minor things

* zonal/meridional_flux_adjust: use scalar u/v_new

* declare vp,up,h_tmp on gpu

* remove h update

* continuity_PPM: minimize mapping stmts

* zonal_flux_adjust: minimize mapping stmts

* set_zonal_bt_cont: minimize mapping stmts

* merional_flux_adjust: minimize mapping stmts

* set_merid_bt_cont: minimize mapping stmts

* zonal/meridional_flux_adjust: tmp vars duhdu/dvhdv 3d -> scalar

* separate alloc of private variables for gcc

* u/vh_aux: 3d->2d

* target teams loop recognizable by amdflang

* Continuity CS outside of init

This moves the Continuity CS to the dycore init function.  For some
reason, this avoids an answer change with CPU.  (Possibly because alloc
inside of a function doesn't quite match the CS outside of it?)

A few minor data transfers are also added to fix up differences in the
chksum log output.

* Continuity: Add locality to do concurrent

Do concurrent inside of !$omp target teams loop seems to fail standard
openmp tests if locality is not correctly set.

This patch adds the correct locality to the four `!$omp target teams
loop` directives.

The domore argument has also been removed, and replaced with a
`.not.any(do_i(:))` test.

---------

Co-authored-by: Marshall Ward <marshall.ward@noaa.gov>

Merge in latest changes from dev/gfdl